The present invention relates to a method and system for tracking a target and, more particularly, to such a method and system that uses spectral images of a scene including the target in order to track the target.
FIG. 1 shows a battlefield scene including an enemy tank 10 and three trees 12. FIG. 2 is a schematic diagram of a prior art system that includes an unmanned aircraft 20 that may be, for example, a surveillance drone that is used by an operator 14 to locate and track tank 10, or alternatively a guided missile that is guided by operator 14 towards tank 10 to destroy tank 10. Aircraft 20 includes an imaging mechanism 22, such as a video camera, for acquiring digital images of the scene of FIG. 1, and a processor 24 for designating and tracking image pixels that correspond to tank 10, as described below. (Aircraft 20 also includes other components, such as a navigation system for determining the location of aircraft 20 and a communication system for relaying the digital images to operator 14 via a wireless communication channel 18 and for relaying commands from operator 14 to aircraft 20. These other components, not being germane to the present invention, are not shown in FIG. 2.) The images acquired by imaging mechanism 22 are panchromatic: each image is a rectangular array of pixels, with each pixel being a numerical representation of a value of a corresponding intensity of light received by imaging mechanism 22, in a single wide or narrow spectral band, from a corresponding point in the field of view of imaging mechanism 22.
Aircraft 20 transmits the images acquired by imaging mechanism 22 to a console 16 that is used by operator 14 to operate aircraft 20. These images are displayed on a video terminal that forms part of console 16. When operator 14 sees an image that includes pixels corresponding to tank 10, operator 14 designates those pixels, using a conventional mechanism such as a mouse to pick the image coordinates of one or more pixels that correspond to tank 10. These coordinates are transmitted to aircraft 20 via communication channel 18. In the subsequent discussion, the image in which operator 14 designates the pixels corresponding to tank 10 is called the “first” image, because this is the image that guides subsequent processing by processor 24. Because drone 20 is moving relative to tank 10, whether or not tank 10 itself is moving, the pixels corresponding to tank 10 in subsequent images are not, in general, the same pixels as the pixels corresponding to tank 10 in the first image. Therefore, processor 24 uses a cross-correlation technique to identify pixels corresponding to tank 10 in subsequent images.
FIG. 3 shows a panchromatic image 30 and a reference window 32 within image 30. Image 30 is an I×J rectangular array of pixels that are indexed by indices i and j, iε[1,I], jε[1,J]. With each pixel is associated a respective intensity value. Window 32 is a H×K rectangular subset of the pixels of image 30 that includes the pixels corresponding to the target. Typically, I and J are powers of 2, such as 512 or 1024, and H and K are small odd integers, such as 5 or 7.
Processor 24 constructs reference window 32, in the first image 30, that is centered on the image coordinates picked by operator 14 as corresponding to tank 10. Processor 24 then cross-correlates reference window 32 with the next image 30 by constructing a correlation function Corr(i,j). The argument of this correlation function is a pair of pixel indices (i,j). For each pixel (i,j) in the next image 30 for which reference window 32 can be centered on that pixel while still being contained within image 30 (i.e., for all the pixels except for a margin └H/2┘ pixels wide at the top and bottom of image 30 and └K/2┘ pixels wide on the left and right of image 30), Corr(i,j) is defined as:
                              Corr          ⁡                      (                          i              ,              j                        )                          =                                            ∑                                                h                  =                                      1                    :                    H                                                  ,                                  k                  =                                      1                    :                    K                                                                        ⁢                                          x                                  h                  ,                  k                                            ⁢                              y                                                      i                    -                                          ⌊                                              H                        /                        2                                            ⌋                                        -                    1                    +                    h                                    ,                                      j                    -                                          ⌊                                              K                        /                        2                                            ⌋                                        -                    1                    +                    k                                                                                                                          ∑                                                      h                    =                                          1                      :                      H                                                        ,                                      k                    =                                          1                      :                      K                                                                                  ⁢                                                x                                      h                    ,                    k                                    2                                ⁢                                                      ∑                                                                  h                        =                                                  1                          :                          H                                                                    ,                                              k                        =                                                  1                          :                          K                                                                                                      ⁢                                      y                                                                  i                        -                                                  ⌊                                                      H                            /                            2                                                    ⌋                                                -                        1                        +                        h                                            ,                                              j                        -                                                  ⌊                                                      K                            /                            2                                                    ⌋                                                -                        1                        +                        k                                                              2                                                                                                          (        1        )            where xh,k is the intensity of the (h,k) pixel of reference window 32, yh,k is the intensity of the (h,k) pixel of image 30, and, for a real number a, the notation └a┘ represents the largest integer that is less than or equal to a. (Note that the correlation function Corr is called “C” in the“Proof of Concept” section below.) With the indicated normalization, and given that all intensities xk,l and yk,l are non-negative, Corr(i,j) can take on values only in the interval [0,1].
Alternatively, and preferably, Corr(i,j) is defined as:
                              Corr          ⁡                      (                          i              ,              j                        )                          =                                            ∑                                                h                  =                                      1                    :                    H                                                  ,                                  k                  =                                      1                    :                    K                                                                        ⁢                                          (                                                      x                                          h                      ,                      k                                                        -                                      x                    _                                                  )                            ⁢                              (                                                      y                                                                  i                        -                                                  ⌊                                                      H                            /                            2                                                    ⌋                                                -                        1                        +                        h                                            ,                                              j                        -                                                  ⌊                                                      K                            /                            2                                                    ⌋                                                -                        1                        +                        k                                                                              -                                      y                    _                                                  )                                                                                        ∑                                                      h                    =                                          1                      :                      H                                                        ,                                      k                    =                                          1                      :                      K                                                                                  ⁢                                                                    (                                                                  x                                                  h                          ,                          k                                                                    -                                              x                        _                                                              )                                    2                                ⁢                                                      ∑                                                                  h                        =                                                  1                          :                          H                                                                    ,                                              k                        =                                                  1                          :                          K                                                                                                      ⁢                                                            (                                                                        y                                                                                    i                              -                                                              ⌊                                                                  H                                  /                                  2                                                                ⌋                                                            -                              1                              +                              h                                                        ,                                                          j                              -                                                              ⌊                                                                  K                                  /                                  2                                                                ⌋                                                            -                              1                              +                              k                                                                                                      -                                                  y                          _                                                                    )                                        2                                                                                                          (        2        )            where {right arrow over (x)} is all average of xh,k, either over image 30 or, alternatively, over a suitable portion of image 30, such as reference window 32 itself, and similarly for yh,k. Corr(i,j), as defined in equation (2), can take on values in the interval [−1,1]
Ideally, Corr(i,j) would be equal to 1 when window 32 is centered on the pixels of image 30 that correspond to the target, and less than 1 elsewhere. This is almost never the case, for a variety of reasons. Among these reasons are noise in image 30 and window 32, and the fact that if image 30 is not the source image of window 32, then image 30 almost always is acquired from a different point of view, relative to the scene and the target than the source image of window 32, because of movement of both drone 20 and the target relative to the scene. Therefore, the location of the pixels of the next image 30 that correspond to the target is taken to be the set of pixels centered on the pixel (i,j) for which Corr(i,j) is maximized. Only pixels (i,j) for which Corr(i,j) exceeds an adaptively determined threshold are considered. Furthermore, to avoid false positives, only a portion of the next image 30 in which the target is likely to be found a priori is considered.
This cross-correlation procedure is repeated for subsequent images 30, to track the target in subsequent images 30. As needed, reference window 32 is replaced with a new reference window based on the pixels corresponding to the target in the most recently acquired image 30.
Recently, a new technology, whose most notable application to date has been in remote sensing, has matured. In this new technology, a scene is imaged in several spectral bands. Such imaging is known generally as “spectral imaging”. If a small number (fifteen or fewer) of relatively broad spectral bands (for example, red, green and blue bands) are used, this technique is referred to as “multispectral imaging”. If a large number of relatively narrow bands are used, this technique is referred to as “hyperspectral imaging”. The first table (unnumbered) in the “Proof of Concept” section below lists a typical example of 19 adjacent spectral bands that span wavelengths from 435.3 nanometers (blue) to 891.1 nanometers (near IR). The “spectral images” acquired by spectral imaging are three dimensional arrays of intensity values, with each intensity value corresponding to the intensity of one scene pixel in one of the imaged bands. FIG. 4 shows a spectral image 40 that consists of L I×J spectral layers 42. Three spectral layers 42 are shown explicitly. The presence of the remaining spectral layers 42 is indicated by ellipses ( . . . ). Each spectral layer 42 is a panchromatic image in its own right. Thus, spectral image 40 is a parallelepiped array of pixels indexed by three indices i, j and l, with iε[1,I], jε[1,J] and lε[1,L]. Each vertical column of spectral image 40, i e., the set of pixel intensities indexed by a particular index pair (i,j) for all values of l in [1,L], is referred to herein as a “pixel vector”.
Obviously, spectral image 40 includes more information than any of its component spectral layers 42, either considered in isolation or summed along the wavelength (l) axis to form a single wide-band panchromatic image. There is thus a widely recognized need for, and it would be highly advantageous to have, a method of tracking a target that is based on spectral images such as image 40, rather than on panchromatic images such as image 30.